Introduction to Natural-language-Processing (R)

### Load standardpackages
library(tidyverse) # Collection of all the good stuff like dplyr, ggplot2 ect.
library(magrittr) # For extra-piping operators (eg. %<>%)

This session

In this applied session, you will:

1. xxx
2. xxx
3. xxx

Refresher: Basics of String Manupilation

We start by taking a piece of text and turning it into something that carries the meaning of the initial text but is less noisy and thus perhaps easier to “understand” by a computer.

text <- "The Eton-educated, non-binary British Iraqi had always struggled with their identity, until they discovered drag. Yet the 29 year old says the performances come at a high price"

# Transforming to lower case
text %>% str_to_lower()

[1] "the eton-educated, non-binary british iraqi had always struggled with their identity, until they discovered drag. yet the 29 year old says the performances come at a high price"

# Split by '.' (=sentence)
text %>% str_split('\\.')

[[1]]
[1] "The Eton-educated, non-binary British Iraqi had always struggled with their identity, until they discovered drag"
[2] " Yet the 29 year old says the performances come at a high price"                                                 

text %>% str_replace_all('o', 'O')

[1] "The EtOn-educated, nOn-binary British Iraqi had always struggled with their identity, until they discOvered drag. Yet the 29 year Old says the perfOrmances cOme at a high price"

# Split by ' ' (=word)
text %>% str_remove_all('[[:punct:]]') %>% str_split(' ') %>% unlist()

 [1] "The"          "Etoneducated" "nonbinary"    "British"      "Iraqi"        "had"          "always"       "struggled"    "with"        
[10] "their"        "identity"     "until"        "they"         "discovered"   "drag"         "Yet"          "the"          "29"          
[19] "year"         "old"          "says"         "the"          "performances" "come"         "at"           "a"            "high"        
[28] "price"       

text %>% str_to_lower() %>% str_remove_all('[[:punct:]]') %>% str_split(' ') 

[[1]]
 [1] "the"          "etoneducated" "nonbinary"    "british"      "iraqi"        "had"          "always"       "struggled"    "with"        
[10] "their"        "identity"     "until"        "they"         "discovered"   "drag"         "yet"          "the"          "29"          
[19] "year"         "old"          "says"         "the"          "performances" "come"         "at"           "a"            "high"        
[28] "price"       

The R NLP ecosystem

• Most language analysis approaches are based on the analysis of texts word-by-word.
• Here, their order might matter (word sequence models) or not (bag-of-words models), but the smallest unit of analysis is usually the word.
• This is usually done in context of the document the word appeared in. Therefore, on first glance three types datastructures make sense:

1. Tidy: Approach, where data is served in a 2-column document-word format (e.g., tidytext)
2. Token lists: Creation of special objects, saved as document-token lists or corpus (e.g., tm, quanteda)
3. Matrix: Long approach, where data is served as document-term matrix, term-frequency matrix, etc.

• Different forms of analysis (and the packages used therefore) favor different structures, so we need to be fluent in transfering original raw-text in * These formats, as well as switching between them. (for more infos, check here).

Tidy Text Formats

• While there exist other ecosystems to do txt analysis (e.g., tm, quanteda), I will here almost exclusively use tidytext, which is very simple yet powerful, very well documented, and works very neathly with tidymodels and the rest of the tidyverse ecosystem.

library(tidytext)

• While we will for later applications we will use different formats, we here will limit ourselves to word token, which can do most of the simple jobs.
• Here, we apply tidy principles to text, make word-token per document our unit of analysis.
• Therefore, every row repreesents a word per document. This sounds like a lot of redundancy, but makes it very easy to work with compared to more complez matrix and list formats. Here, we can do our usual sumarries and visualizations pretty much out-of-the-box.

text_tbl <- tibble(id = 1, text = text)

# We now unnest the tokens. Notice it is by default deleting all punctuation and transforming the text to lower chars.
text_tidy <- text_tbl %>% unnest_tokens(word, text, token = 'words')

• Overall, in NLP we are trying to represent meaning structure.
• That means that we want to focus on the most important and “meaning-bearing elements” in text, while reducing noise.
• Words such as “and”, “have”, “the” may have central syntactic functions but are not particularly important from a semantic perspective.

# Defining stopwords
stop_words

text_tidy %<>%
  anti_join(stop_words, by = 'word')

text_tidy

# We now unnest the tokens. Notice it is by default deleting all punctuation and transforming the text to lower chars.
sentences_tidy <- text_tbl %>% unnest_tokens(word, text, token = 'sentences')

sentences_tidy

Your turn!

Take the following text and transform it into a list of lists with with each element being a tokenized sentence. Remove stopwords, lower all tokens and keep only alpha-numeric tokens.

I’ve been called many things in my life, but never an optimist. That was fine by me. I believed pessimists lived in a constant state of pleasant surprise: if you always expected the worst, things generally turned out better than you imagined. The only real problem with pessimism, I figured, was that too much of it could accidentally turn you into an optimist.

source: https://www.theguardian.com/global/2019/nov/21/glass-half-full-how-i-learned-to-be-an-optimist-in-a-week

data <- read_csv('https://sds-aau.github.io/SDS-master/00_data/xxx.csv') 

Endnotes

Your turn

Please do Exercise 1 in the corresponding section on Github.

Packages & Ecosystem

• tidygraph here
• ggraph here

Suggestions for further study

• DataCamp (!All courses have somewhat outdated ecosystems)
  • Network Analysis in R: Good for some of the basics

Session Info

sessionInfo()